Camera calibration with lenticular arrays

ABSTRACT

The present disclosure is directed to an approach to solve camera calibration from a single picture of a simply structured light field created by viewing a carefully designed lenticular sheet. The lenticular sheet has different colors visible at different relative orientations and gives linear constraints on the K-1?R matrix which can be decomposed to give the rotation and calibration. The method uses geometric cues from hue measurements and does not require a correspondence between a point in the image and a point on the calibration pattern.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority benefit of U.S. Provisional ApplicationSer. No. 62/150,028, filed on Apr. 20, 2015, the entire content of whichis incorporated herein by reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH & DEVELOPMENT

This invention was made with government support under IIA1355406,awarded by National Science Foundation. The government has certainrights in the invention.

FIELD OF THE DISCLOSURE

The field of the disclosure relates generally to camera calibration and,more specifically, to calibration without correspondence from astructured light field.

BACKGROUND OF THE DISCLOSURE

Camera calibration defines the relationship between an image and thescene that the image depicts. Calibration characterizes the photometricand geometric properties of the camera, that define, respectively howpixels of the camera report color and intensity of the scene, and wherescene elements appear on the image.

Most common calibration approaches start with an image of an object withknown 3-D geometry, or several images of an object with known 2-Dgeometry, find correspondences between points on the object and pointsin the image, and use these to solve for the camera geometry.

There are some approaches that are not based on identifying exactcorresponding points in a scene. Calibration patterns that consist ofpatches of parallel lines can be used for intrinsic camera calibrationand extrinsic calibration (including rotation and translation relativeto other cameras). Approaches that do not require correspondence betweenspecific lines are based on seeing the orientation and spacing of thoseparallel lines on the image and include those based on the prismaticline constraint, and an approach using the Radon transform as afiltering operator. In some circumstances, there are scenes that havelarge numbers of parallel lines with a known orientation, such asvertical edges of buildings or plumb-lines; the orientation and positionof those lines in an image provide constraints for both intrinsic andextrinsic calibration even without matching the pixel to a specific linein the world.

Other approaches use textures for calibration which assumes that naturalurban textures have low rank (for example, grids of windows on theexterior of a building). Using this assumption, calibration and lensdistortion that minimizes the rank of the textures in the image, usingall pixels, not just point locations of geometric points is solved for.

BRIEF DESCRIPTION OF THE DISCLOSURE

In one embodiment of the present disclosure, a method of calibrating acamera is disclosed. The method comprises creating a light field with alenticular array; and, using a single picture of the light field tocalibrate the camera.

In another embodiment of the present disclosure, a method of creating alight field is disclosed. The method comprises using a calibrationobject, wherein the calibration object is based on a lenticular arraycomprising at least one sheet; and, focusing light onto a back side ofthe at least one sheet to create the light field.

In yet another embodiment of the present disclosure, a method ofcalibrating a camera for an augmented reality application is disclosed.The method comprises using a calibration object to create a color-codedlight field, wherein the calibration object is based on at least onelenticular array; estimating a focal length of the camera and a relativeorientation and position of the object from a single image; and,calibrating the camera.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1D are exemplary embodiments of a lenticular array consistingof a set of cylindrical lenses designed to focus parallel light raysonto a back-plane in accordance with the present disclosure.

FIG. 2 is an exemplary embodiment of a notation in a coordinate systemof a lenticular patch where x is the direction of the dominant axis,V_(hue) is a ray seen for each hue, and n_(hue) is the derivations inaccordance with the present disclosure.

FIG. 3A is an exemplary embodiment of a function of the amount of noiseshowing sensitivity to noise in rotation estimation for all axes inaccordance with the present disclosure. FIG. 3B is an exemplaryembodiment of a function of the amount of noise showing sensitivity tonoise in focal length estimation in accordance with the presentdisclosure. FIG. 3C is an exemplary embodiment of a noise profileshowing the quartiles of angular error introduced to {right arrow over(n)}_(hue) for a given standard deviation of noise in accordance withthe present disclosure.

FIG. 4A is an exemplary embodiment of a rotation estimation showing thefirst, second, and third quartiles of errors in accordance with thepresent disclosure. FIG. 4B is an exemplary embodiment of a focal lengthestimation showing the first, second, and third quartiles of errors inaccordance with the present disclosure.

FIGS. 5A-5G are exemplary embodiments of designs created by changing theorientation and positions of the lenticular arrays to make a calibrationobject in accordance with the present disclosure. FIG. 5H is anexemplary embodiment of an estimation error of the rotation around eachaxis in accordance with the present disclosure. FIG. 5I is an exemplaryembodiment of a focal length error in accordance with the presentdisclosure.

FIGS. 6A-6D are exemplary embodiments of calibrated images at variousposes and focal lengths in accordance with the present disclosure.

FIGS. 7A and 7B are exemplary embodiments of images taken with longfocal length lenses in accordance with the present disclosure.

FIG. 8A is an exemplary embodiment of an image showing the focus infront of the calibration object in accordance with the presentdisclosure. FIG. 8B is an exemplary embodiment of an image showing thefocus at the calibration object in accordance with the presentdisclosure. FIG. 8C is an exemplary embodiment of an image showing thefocus behind the calibration object in accordance with the presentdisclosure.

FIG. 9A is an exemplary embodiment of a plot showing the sensitivity torotation when the focal length estimates as rotations are kept constantin accordance with the present disclosure. FIG. 9B is an exemplaryembodiment of a plot showing the sensitivity to focal length inaccordance with the present disclosure. FIG. 9C is an exemplaryembodiment of a plot showing the sensitivity to rotation when the focallength estimates as rotations are kept constant in accordance with thepresent disclosure. FIG. 9D is an exemplary embodiment of a plot showingthe sensitivity to focal length in accordance with the presentdisclosure.

FIGS. 10A and 10B depict an exemplary embodiment of a calibration objectmade from three lenticular arrays in accordance with the presentdisclosure.

FIGS. 11A-11D depict views of lenticular arrays in accordance with thepresent disclosure.

FIG. 12A depicts an image of an exemplary calibration object inaccordance with the present disclosure. FIG. 12B depicts an exemplaryembodiment of observed hue measurement at the yellow circles of FIG. 12Ain accordance with the present disclosure.

FIGS. 13A-13C depict various hue measurements in accordance with thepresent disclosure.

FIGS. 14A and 14B depict exemplary prediction errors in accordance withthe present disclosure.

FIGS. 15A-15D depict exemplary rotation estimations and rotationperformances in accordance with the present disclosure.

FIGS. 16A and 16B depict exemplary embodiments of focal lengthestimations and performance in accordance with the present disclosure.

FIGS. 17A-17D depict exemplary initial translation estimations andperformances in accordance with the present disclosure.

FIGS. 18A-18H depict exemplary embodiments of single frame estimationsof the focal length and object rotation estimates in accordance with thepresent disclosure.

FIG. 19 depicts an exemplary embodiment of the focal length error thatarises from mis-estimating a field in accordance with the presentdisclosure.

FIG. 20 depicts an exemplary embodiment of the focal length estimatesand performance in accordance with the present disclosure.

FIGS. 21A-21I depict exemplary embodiments of focal length estimationresults in three frames of a video over a calibration object inaccordance with the present disclosure.

FIG. 22 depicts an exemplary embodiment of an anchor point localizationHRF prediction accuracy in accordance with the present disclosure.

FIGS. 23A and 23B depict exemplary embodiments of the space of hues witha constant saturation and value on a manifold in RGB space.

FIGS. 24A and 24B depict exemplary embodiments of RGB measurements inaccordance with the present disclosure.

FIGS. 25A and 25B depict exemplary embodiments of RGB measurements inaccordance with the present disclosure.

FIGS. 26A-26F depict exemplary embodiments of focal length, rotation andtranslation estimates in accordance with the present disclosure.

DETAILED DESCRIPTION OF THE DISCLOSURE

The present disclosure is directed to an approach to solve cameracalibration from a single picture of a simply structured light fieldcreated by viewing a carefully designed lenticular sheet. The lenticularsheet has different colors visible at different locations and giveslinear constraints on the K⁻¹R matrix which can be decomposed to givethe rotation and calibration. The pixel locations that are measured arelinear and the hue does not require a correspondence between the imageand a point on the calibration pattern. Sensitivity to noise, the designof a lenticular calibration sheet, and experimental results from aphysical prototype pattern were studied.

In some embodiments, this approach to partial geometric calibration of acamera uses a passive calibration object that creates a speciallystructured light field. In some embodiments, the calibration object isbased on a lenticular array. These are sheets of plastic which arecomprised of a large set of parallel cylindrical lenses. These lensesfocus light onto the back of the plastic sheet. For example, commonchildren's toys exploit this by interleaving multiple pictures behindthe lenticular array so that the apparent pattern changes as the arrayis rotated. In another example, some modern televisions (TVs) use alenticular layer in order to show different pictures in differentdirections in order to project 3-D TV “without any glasses.”

In some embodiments, a method of calibrating a camera is disclosed. Themethod comprises creating a light field with a lenticular array; and,using a single picture of the light field to calibrate the camera. Insome embodiments, the camera is calibrated without any correspondencefrom a structured light field.

In some embodiments, the lenticular array comprises at least one sheet.In some embodiments, the at least one sheet comprises a plurality ofcolors. In some embodiments, the plurality of colors are visible at aplurality of locations on the sheet. In some embodiments, the sheetcomprises plastic. Further, in some embodiments, the at least one sheetcomprises a plurality of parallel cylindrical lenses.

In another embodiment, a method of creating a light field is disclosed.The method comprises using a calibration object, wherein the calibrationobject is based on a lenticular array comprising at least one sheet;and, focusing light onto a back side of the at least one sheet to createthe light field. In some embodiments, the light field is not astructured light field. In some embodiments, a plurality of lensesfocuses the light onto the back side of the at least one sheet.

In some embodiments, not only is the camera calibrated but the lightfield also is used to do pose estimation. In some embodiments, thecamera is calibrated without information from a correspondence between apoint on an image and a point on a calibration object. In someembodiments, a printed pattern is adhered to the back of a lenticulararray and comprises a plurality of interleaved colors. In someembodiments, the plurality of colors are visible at a plurality oforientations relative to the lenticular array.

FIGS. 1A-1D are exemplary embodiments of a lenticular array consistingof a set of cylindrical lenses designed to focus parallel light raysonto a back-plane in accordance with the present disclosure. As shown inFIG. 1A, if this back plane has different colors arranged behind eachlens, then the apparent color of the lenticular array changes based onviewing angle. This pattern is repeated at every location of thelenticular array, so each lenticule is viewed as a light source that hasa different color projecting in each direction (shown in FIG. 1B). Thislenticular array creates a light field structured so that the color ofeach ray depends on its direction (shown in FIG. 1C). The image from apinhole camera in this light field will have colors that change acrossits field of view, and derives simple constraints that allow this imageto be used for camera calibration without solving for imagecorrespondences (shown in FIG. 1D). As shown in FIGS. 1A-1D, theinterleaved pictures behind each lenticular lens are a color spectrum.This creates a light field where the color of a pixel dependsfundamentally on the direction of the ray viewed by that pixel. This isquite different than most imaging situations; for example, in Lambertianscenes, it is the set of rays that come from a single world point thathave the same appearance. A camera that captures an image in this lightfield will see a color pattern that varies across the image.

These images have a beneficial relationship to the parameters of acamera's rotation and intrinsic calibration. Specifically, the morequickly the color changes across the lenticular array in the image, theshorter its focal length. The constraints are only slightly morecomplicated in a full three dimensional setting, and the contributionsof this disclosure are to formally derive those constraints, to give aninitial exploration of the sensitivity to noise of these constraints,and to experimentally demonstrate their feasibility with a lenticularcalibration object.

The following three points are particularly compelling: (i) lenticularsheets are a cheap commodity item and the appropriate back-planetextures can be printed on commodity printers, so this approach isbroadly feasible even in application domains where cost is a factor;(ii) the constraints on the intrinsic parameters and the rotation arelinear constraints based only on the pixel location and observed colorthey do not require correspondence between a pixel and a location on thecalibration grid, so it is easy to use all observed pixels asmeasurements; and, (iii) when a camera is translated within this lightfield, the images remain exactly the same.

Structured Light Field Patterns Using Lenticular Arrays

In some embodiments, a structured light field based on a lenticulararray with a back-plane pattern that varies in hue is defined. Thisleads to an object whose apparent color smoothly varies as it isrotated. The observed hue depends on how far the viewing angle isrotated around the dominant axis of the lenticular array. FIG. 2describes a notation that is used, in the coordinate system of thelenticular patch x is the direction of dominant axis, V_(hue) is anexemplary ray observed at a given color and n_(hue)=V_(hue)× x is thesurface normal to the plane containing all rays of that color.

At shallow viewing angles, the mapping between the hue and the viewingangle is more likely to be incorrect because the lenticular lenses nolonger focus rays perfectly and there is more likely to be specularreflections off the surface of the lens. In making small fiducialmarkers from lenticular patches, it is described how to experimentallycalibrate a physical prototype to solve for the V_(hue) and n_(hue) thatcorrespond to a given color, and characterize the set of angles wherethe hue to direction mapping is consistent.

Derivation of Calibration Constraints from Lenticular Patterns

In some embodiments, a pinhole camera views a lenticular pattern. Therotation of that lenticular pattern relative to the camera is R, and thecalibration matrix of the camera is K. The basic constraint is that aray observing a hue must be perpendicular to n_(hue). In the coordinatesystem of the camera, a pixel {right arrow over (ρ)} captures lighttraveling along a ray r that depends on the calibration matrix K as:{right arrow over (r)}=K ⁻¹ρ  Equation 1.0

In the coordinate system of the lenticular pattern, this ray is RK⁻¹ρ,so when observing a particular hue, it must satisfy the constraint:(RK ⁻¹ρ)·{right arrow over (n)} _(hue)=0.  Equation 1.1

This dot-product is written in matrix notation,(RK ⁻¹ρ)^(T) n _(hue)=0,  Equation 1.2

rewrite the transpose to getρ^(T) K ^(−1T) R ^(T) {right arrow over (n)} _(hue)=0.  Equation 1.3

collect terms to getρ^(T)(K ^(−1T) R ^(T)){right arrow over (n)} _(hue)=0.  Equation 1.4

and rewrite the center matrix as:ρ^(T)(RK ⁻¹)^(T) {right arrow over (n)} _(hue)=0  Equation 2.0

In this equation, p is a pixel location and {right arrow over (n)}_(hue)is a vector constraint because the lenticular pattern is a particularhue only when viewed from some directions. M is the combination of theunknown calibration and rotation so M=(RK⁻¹)^(T). If consideringmeasuring the hue at many pixels, each pixel ρ_(i) and the constraintfrom measuring its hue {right arrow over (n)}_(hue) combine to create alinear constraint on the unknown M:ρ_(i) ^(T) M{right arrow over (n)} _(hue)=0  Equation 3.0

Given an image, it can measure the hue at many pixel locations, uselinear least squares to solve for M, and then use the QR-decompositionto extract the estimates of K and R.

Translation of the lenticular pattern and the position on the lenticularpattern of the observed point appear nowhere in this set of constraints,so it does not require correspondence between the pixels and thecalibration pattern. However, if all observed points are from a singlelenticular sheet with the same dominant orientation, then the n_(hue) isrestricted to lie along pencil of planes (all planes that include thedominant direction x).

Enforcing Simplified Calibration Matrices

In some embodiments, it may be desirable to enforce specific propertieson the matrix K, such as a known image center, square pixels, and zeroskew (leaving only the focal length unknown). In this case, the onlyfree variables are the focal length f and the rotation which isparameterized with Rodrigues vector ρ. Then, one can solve for thesevariables minimizing the squared error function, summed over all pixelsi:g(f,β)=Σ_(i)∥ρ_(i) ^(T) K(f)R(β){right arrow over (n)} _(huei)∥₂².  Equation 4.0

where K(f) is calibration matrix that has focal length f and assumesthat the camera principal point is at the center of the image (c_(x),c_(y)):

$\begin{matrix}{{{K(f)} = \begin{pmatrix}f & 0 & {cx} \\0 & f & {cy} \\0 & 0 & 1\end{pmatrix}},} & {{Equation}\mspace{14mu} 5.0}\end{matrix}$

and R(p) is the rotation matrix corresponding to the Rodrigues vector ρ.

Single Image Camera Calibration with Lenticular Arrays for AugmentedReality

In another embodiment of the present disclosure, a method of calibratinga camera for an augmented reality application is disclosed. The methodcomprises using a calibration object to create a color-coded lightfield, wherein the calibration object is based on at least onelenticular array; estimating a focal length of the camera and a relativeorientation and position of the object from a single image; and,calibrating the camera.

In some embodiments of the present disclosure, an approach to geometriccalibration of a camera that requires a single image of a calibrationobject that may lie within a larger scene is disclosed. An exemplaryembodiment of the calibration object is shown in FIGS. 10A and 10B. Eacharray has an observed color that changes depending on the viewing angle.In FIG. 10A, when viewed from a far distance, the arrays have relativelyconsistent colors because they are being viewed from approximately thesame angle. In FIG. 10B, a wide angle view from a closer distance hassignificant color variation because the direction from the camera todifferent parts of the object varies substantially.

This calibration object is based on several lenticular arrays. In someembodiments, a lenticular array is defined as a sheet of plastic whichis comprised of many tiny parallel cylindrical lenses. FIGS. 11A-11Ddepict how these lenses focus parallel rays of light onto an interleavedpattern on the back of the lenticular array. As a result, for differentviewpoints, the lenticular array has different appearances.

In some embodiments, a lenticular pattern is constructed based on apattern of interleaved colors. This creates an apparent hue dependent onthe relative incident angle of light rays viewing the lenticular array.For a perspective camera viewing a planar surface, pixels may havediffering viewing angle and therefore will measure a different hue.Therefore, as seen in FIGS. 11C and 11D, a camera with a wide field ofview would see many different hues, while a camera with a narrow fieldof view would see fewer hues. This fundamental relationship between acolor-coded lenticular array and a camera provides a novel geometricconstraint to calibrate a camera.

In accordance with the present disclosure, the inventors havesurprisingly been able to: allow lenticular patterns to constrain thecamera focal length; created an approach to correct manufacturingproblems of alignment and stretching that make relationship betweencolor and angle vary across a lenticular array; and, experimented with aphysical instantiation of a prototype calibration object, showingcalibration accuracy in different settings and a complete end-to-endaugmented reality demonstration with a variable zoom video.

In Augmented Reality (AR), one seeks to render digital content on top ofa video feed of a scene to digitally enhance the physical world. Inorder to properly project digital elements into the video feed, therelative pose of the digital element and the camera must be known, andthe camera calibration must be known.

Calibration methods which require multiple images of an object indifferent poses or calibration objects with substantial variation in all3 dimensions may not be appropriate for all applications.

As disclosed herein, the approaches calibrate every image independentlyusing a flat calibration object. This object is based on a lenticularpattern. Lenticular arrays and their 2-D counter-part, microlens arrays,give geometric constraints on the incident angle of light rays viewingthe arrays. The present disclosure addresses the problem of jointintrinsic and extrinsic camera calibration needed in AR applicationswhere cameras may change their zoom to keep track of an object. Such anapproach is necessary to integrate AR with new commercial systems thatsell Pan-Tilt-Zoom cameras that automatically track a radio tag, butwhich do not have encoders on their zoom lenses that tag imagery withthe zoom level.

AR Calibration Object

An exemplary calibration object disclosed herein is inspired by thelenticular array used to estimate object rotation. In some embodiments,three lenticular arrays are mounted perpendicular to each other on aplane, where the two flanking arrays have the same orientation, butorthogonal to the middle array. These arrays are orthogonal so that anyrotation of the calibration object creates a change; when the object isoriented as shown in FIG. 10A, rotation around the horizontal axiscauses the two edge arrays to change color, while rotating around thevertical axis causes the central part to change color. Small blackstrips are added to make it easier to distinguish the three arrays whenthey are oriented so that their colors are similar.

Calibrating a Color Coded Lenticular

The relationship of the apparent viewing angle to the observed colordepends on the relative angle of the lenticular array, in particular therotation around the axis of the lenticular lenses. The relationshipbetween this rotation and observed color is captured in the Hue ResponseFunction (HRF), which is a 1-to-1 relationship for incident angles of upto 40 degrees (after which the colors repeat).

For a lenticular array, the hue response function varies across thearray. As shown in FIGS. 12A and 12B, a picture of the calibrationobject taken from far away with a very long focal length lens, gives afield of view of 1 degree. Therefore, the color difference observed atthe two circles in FIG. 12A are substantial. When measuring this colordifference as the calibration array rotates, a consistent shift is seen.This is due to the challenges of manufacturing and printing the colorcoded pattern that sits behind each lenticular lens. For this lenticulararray that has 2 lenticular lenses per millimeter, if the backplane isstretched about 0.1 mm extra over the course of this array, this causesthe observed color shift.

To address this challenge, while still employing standard manufacturingprocesses for lenticular arrays, the HRF was calibrated, the functionthat relates hue to orientation, at regular intervals in the localreference frame of the arrays. The corners of the rectangular lenticulararrays were used as anchor points, and for each image a homography wascomputed mapping the observed lenticular patterns to a canonicalcoordinate system. The process is illustrated in FIGS. 13A-13C. Thiscalibration object is placed on a controlled rotation mount and rotatedthrough 1 degree increments. For each calibration point, the angle atwhich that calibration point is viewed was recorded (which may varyacross the calibration grid because of perspective effects), and the huefor that angle was measured. The result of this is a curve like thoseshown in FIG. 12B for each of the calibration points. This process wasrepeated separately for the center and each of the two side lenticulararrays that make up the calibration object.

The next section derives how images of this calibration object canexploit the measured colors for additional geometric cues to the pose ofthe calibration object and the focal length of the camera. Whenconverting the observations of the object into geometric constraints,the corners of the array were found to compute the homography, and thecolors at these same grid locations were sampled.

Calibration Constraints

Calibration constraints that arise from observing the lenticularcalibration object are discussed herein. The overall constraints for theimaging geometry start with a pixel 1) that observes the lenticulararray. In the coordinate system of the camera, a pixel 13 captures lighttraveling along a ray r that depends on the calibration matrix K asdefined in Equations 1.0-2.0 discussed above.

In some embodiments, the following optimization to get an initialestimate for R and K by parameterizing K by its focal length f isdisclosed:

${{K(f)} = \begin{pmatrix}f & 0 & {xo} \\0 & f & {yo} \\0 & 0 & 1\end{pmatrix}},$

In K(f), pixels are assumed to be square and x₀ and y₀ are assumed to bethe center of the image.

Algorithm

In some embodiments, an algorithm follows the following steps to get aninitial estimate of the calibration object pose and camera focal length:

1. Find the four corners of the lenticular calibration object.

2. Solve for the homography to map image coordinates onto objectcoordinates.

3. Measure the hue at each grid point on the homography, and use thesehue measurements to solve for an initial estimate of the rotation andthe focal length.

4. Given that rotation and focal length, use lenticular marker basedconstraints introduced to get an estimate of the object translation.

In some embodiments, the initial estimate is refined by minimizing thefollowing cost function:

$\begin{matrix}{{\underset{p,T,f}{\arg\;\min}{\sum\limits_{i}\;\left( {{h\left( {{R(\rho)},T,f,i} \right)} - {hue}_{i}} \right)^{2}}} + {\lambda{{{g\left( {{R(\rho)},T,f} \right)} - p_{i}}}_{2}^{2}}} & (3)\end{matrix}$

where the first term penalizes the difference between hue, which is themeasured hue at grid-point i (of all lenticular arrays), and h(R(p); T;f i), the hue predicted for grid point i when it is projected onto theimage based on camera intrinsic and extrinsic parameters R; T; f, usingthe HRF function calibrated for grid-point i. Here, R is parameterizedvia rodgrigues parameters. The second term measures the spatialreprojection error between the location pi and the predicted locationfor that grid point g(R; T; F; i) based on R; T and f. A relativeweighting function was found empirically to balance hue and positionerror which are measured in very different coordinate systems. In allexperiments, λ was set to 1/4000.

EXAMPLES

The following Examples describe or illustrate various embodiments of thepresent disclosure. Other embodiments within the scope of the appendedclaims will be apparent to a skilled artisan considering thespecification or practice of the disclosure as described herein. It isintended that the specification, together with the Example, beconsidered exemplary only, with the scope and spirit of the disclosurebeing indicated by the claims, which follow the Example.

Example 1—Simulated Camera Calibration

The following shows simulated and empirical results of using lenticulararrays to calibrate a camera. First, the sensitivity of the geometricconstraints to noise, number of sample points, and calibration objectdesign were explored. Then, empirical results of a prototype tocalibrate for example, an exemplary phone and camera are shown. Next, itis shown that the method also works for out of focus images. Lastly, theoptimized rotation and focal length parameters in isolation wereexplored.

A simulator was created to explore the geometric constraints, allowingto change parameters that control: the amount of noise in the measuredhue of a simulated lenticular array; the number of measurements takenfrom a simulated calibration object; and the orientation and relativepositioning of various lenticular arrays that make up a simulatedcalibration object.

This simulator randomly generated a known position and orientation forthe virtual calibration object that is modeled to be 110 mm tall and 70mm wide. This object can appear anywhere in the field of view of avirtual camera from 150 to 850 mm away. In addition, the virtualcalibration object cannot be rotated more than 30° around any of itsaxes. With a randomly generated position and orientation, the simulatorprojected the object from a camera's 3-D space onto an image plane. Thepinhole geometry was a 1/3.2″ format image sensor at a default 4.1 mmfocal length. This image was used to optimize the measured geometricconstraints to get a rotation and focal length estimation. Theseestimations were compared against the true simulated conditions to gaugethe performance of the derived geometric constraints.

Sensitivity to Noise

The only measurement of the calibration system was the hue observed fromthe lenticular array at a given pixel location. Therefore, a source oferror could be in measuring an incorrect hue. In terms of geometricconstraints in previous sections, this error manifested as an improperdirection of {right arrow over (v)}_(hue), and thus {right arrow over(n)}_(hue) Therefore, to simulate the geometric effects of measurementnoise, normally distributed aberrations to the direction of {right arrowover (v)}_(hue) were introduced. These aberrations were created byrandomly choosing a 3-D vector from a Gaussian distribution with a givenstandard deviation, adding that vector to {right arrow over (v)}_(hue),and re-normalizing to again get a unit vector.

This example started with zero noise as a baseline and added a maximumof 0.2 standard deviation of noise to the unit vector, FIG. 3C shows theinsight into the practical effects of adding noise to {right arrow over(v)}_(hue), and then computing {right arrow over (n)}_(hue), whichshowed the angular error in the geometric constraint (the computeddirection of {right arrow over (n)}_(hue)) as a function of the standarddeviation of the added noise. The sensitivity to noise in estimationrotation and focal length as a function of the amount of noise is shownin the rest of the FIG. 3.

The 1st, 2nd (median), and 3rd quartiles of the errors in rotationestimation are shown in FIG. 3A. The angular error for each axis of arotated local reference frame of the calibration object was displayed.The angular error for each axis was measured as the difference (indegrees) of a coplanar ground truth projected axis and the estimatedprojected axis. For all three axes, the trend has higher median amountsof rotational error with wider distributions for increasing amount ofnoise. The x and y axes have a small amount of error more than the zaxis. That was due to the fact that lenticular arrays are directlymeasuring rotation around the x and y axes and not around the z axis.Thus, error in rotation around the z axis manifest as error in theangular error of the x and y axes.

The 1st, 2nd (median), and 3rd quartiles of the errors in estimating thefocal length are shown in FIG. 3B. With more noise, the distribution offocal error got wider. Perhaps more striking, however, is that with morenoise, the focal estimate became smaller. Typically, less than 3 degreesof noise was seen in the {right arrow over (n)}_(hue) measurements,which corresponded to a noise standard deviation of 0.05 in this figure.At that noise level, the experiment showed focal length errors of aboutless than 1% and a rotation error of much less than 1 degree. Even atvery high noise levels, the estimated focal length error was 10% of thetrue focal length and the median estimated rotation has less than 2degrees of error.

Sensitivity to Number of Measurements

Because this calibration approach does not need point correspondences,it was easy to use a large number of measurements to provide redundancy.Thus, how the number of measurements of the calibration object increasedcalibration performance was analyzed.

300 trials of randomly generated calibration object posed with 0.08standard deviation in noise and used an increasing amount ofmeasurements sampled evenly across the calibration object foroptimization were ran. The results are shown in FIGS. 4A and 4B.

The 1st, 2nd (median), and 3rd quartiles of the errors in rotationestimation are shown in FIG. 4A. As more measurements were used,rotation error reduced and become more narrowly distributed. The numberof measurements did not affect the median results for focal estimation,shown in FIG. 4B. However, the estimations became more consistent asmore measurements were used in the optimization.

A measurement represented the hue at one pixel of the image. One can getmore measurements of the calibration object by having the calibrationobject fill more of the image bringing it closer or by using a higherresolution camera. For subsequent experiments, about 30 thousandmeasurements (200×150 pixels) were used, which are feasible to capturewith commodity cameras at reasonable distances.

Sensitivity to Orientation and Relative Position of Lenticular Arrays

The constraints created by observing a single lenticular array were notsufficient to solve for the focal length and camera rotation. To get asystem of equations for two that is not rank deficient, observations ofa lenticular sheet of a different orientation were included. Thus, thecalibration object had two lenticular arrays, which had major axes indifferent directions. Beyond this, there was also the designconsideration of relative positioning of the differently orientedlenticular arrays.

How the relative orientation and placement of lenticular sheets affectedthe estimation accuracy by creating a large set of designs was assessed.Each design is depicted in FIGS. 5A-5G. These show the layout andorientation of the lenticular arrays. For each design, 400 simulationswere run that varied position and rotation, the addition of 0.2 standarddeviation in appearance noise and 30000 measurements. The estimationerror (shown in FIG. 5H) of the rotation around each axis and the focallength (shown in FIG. 5I) were measured.

The design in FIG. 5B was the worst in estimating rotation, because thenon-orthogonal lenticular orientations gave less complementary cuesabout rotation angle. Otherwise, most of the patterns are similar inestimating the rotation angle, and the pattern in design shown in FIG.5C gave the best estimate of the focal length. This might have happenedbecause it has parallel lenticular arrays farthest apart, maximizing theangular difference at which parallel lenticular arrays are viewed andtherefore maximizing the hue difference at different parts of the array.This is important because this hue difference at different parts of thearray is the primary cue for estimating focal length.

Example 2—Evaluation of Physical Prototype

To test the calibration object in practice, a prototype that was 110 mmtall and 70 mm wide was created. It is comprised of two lenticulararrays, one cut smaller and placed directly on top of the other at aperpendicular orientation.

Camera Calibration

The prototype calibration object was tested with two different cameras.First, the images were calibrated with a camera having a 4.1 mm focallength, such as an exemplary phone, at various orientations. Second, theimages were calibrated with an exemplary camera, at various focallengths. Results are shown in FIGS. 6A-6D. For all of the images, theimage shown was used to calibrate the camera, the true and estimatedfocal length in the title (respectively), as well as the estimatedrotation visualized as the local axis of the calibration object.

These results showed rotation estimates that are reasonable, butestimates of the focal length that have up to 40% error. In some images,like FIGS. 6A-6B, direct reflections of overhead fluorescent lights arevisible. Systematic errors in the measured hue, caused by reflections orwhite balance corrections from the cameras are not modeled in thecurrent implementation of the constraints.

For large focal lengths (images taken with telephoto lenses), the methodbreaks down because all the rays from the camera view approximatelyparallel rays, and therefore a lenticular sheet of a particularorientation has the same apparent color. Shown in FIGS. 7A and 7B, aretwo examples of real pictures taken with long focal length lenses thatlead to results where only the surface normal is reasonable.

Calibrating out of Focus Cameras

Using calibration grids requires that all pictures be in focus, tominimize error in locating point correspondences. For close focallengths, it becomes challenging to keep the entire grid in the depth offield, as a variety of orientations and positions of the grid pattern isneeded to achieve strong calibration results. This is not a concern forthis system, as it can calibrate using out of focus images.

An exemplary camera was used and the results of three images that werefocused in front of, at, and beyond the calibration objects are shown inFIGS. 8A-8C. As before, each image is shown with focal length androtation estimations. For all images, this method was able to estimatethe orientation and focal length using the calibration object. In fact,the estimation results were better for the out of focus images. This maybe due to the averaging effect of the hue appearance due to blurring. Asa result, geometric constraint error was propagated more smoothly acrossthe calibration object, resulting in easier optimization.

Exploring Free Parameters

The relationship between the constraints on focal length and rotationwas explored. The same loss function in optimization was used, but arotation or focal length was provided. Therefore, it can introduceincorrect parameters to see how much one parameter is misestimated whenthere is an error in the other.

The errors in the focal length estimation when there is error in therotation around each of the three axes are shown in FIGS. 9A and 9C. Forthese tests, a ground truth rotation by optimizing constraints with theknown focal length of the image was calculated. Then, varying degrees ofrotation around the local axes of the calibration object were used tooptimize for the focal length.

Similarly, introduced error in the focal length in cases where onlyoptimize for the orientation of the calibration object. The angularerror of the rotation estimations as compared to the ground truthrotation are shown in FIGS. 9B and 9D. These error plots are sensitiveto both the design and orientation of the calibration pattern.

A new type of calibration objects that uses lenticular arrays to createstructured light fields is disclosed herein. This calibration patternmust be comprised of lenticular arrays of different orientations andcreating a structured light field where the color of an observed rayrelates to its angle. It leads to a set of constraints for cameracalibration that are linear and do not require direct correspondencebetween pixels and specific locations on the calibration object, so itallows calibration to be based on measurements at very large numbers ofpixels.

Example 3—Sensitivity to Point Tracking

In this example, the ability of an exemplary algorithm and calibrationobject to estimate the object pose and camera focal length was examined.The sensitivity of parts of the algorithm to various intermediateprocessing steps was also explored.

The first stage of the algorithm is tracking the corners of thecalibration object. This is a vital step in most AR pose estimationalgorithms, but it has additional importance in the algorithm because ofmodeling and that the HRF that maps color to angle may vary across thecalibration object. Thus, the sensitivity of the algorithm to errors incorner tracking was evaluated. Second, the accuracy of an exemplaryapproach for estimating the focal length of the camera and rotation andtranslation of the calibration object was evaluated. An error wascharacterized by using a physical prototype with different cameras andwith the calibration object oriented in different directions. Resultsshowed an object added to a video taken with varying focal length, andcompared the realism of the added AR object when there wasn't theability to dynamically estimate the focal length.

Because the lenticular array may not have the same mapping from angle tocolor everywhere, it is important to know where on the calibrationpattern to measure the color in order to look up the correctlocation-specific HRF. Therefore, this approach may be especiallysensitive to estimating the position of the corners of the calibrationobject. This was evaluated by rotating the lenticular array around thevertical axis in one degree increments from −35 to 35 degrees. For eachimage, the following steps were undertaken:

1. Determine four anchor points of the lenticular array,

2. Project the lenticular array into the local reference frame via ahomography

3. Sample the hue from the grid-points of the local reference frameimage.

For each grid point, the angle at which the point was viewed to theangle predicted by the measured hue was computed. To estimate the effectof noise in estimating the lenticular array corners, the anchor pointswere perturbed by 8 pixels in random directions 20 times per image andassess the difference in angle predicted by the HRFs. The scale of onesuch perturbation in the supplementary material is shown in FIG. 22.

FIGS. 14A and 14B show the results. FIG. 14A shows a box and whiskerplot showing the distributions of errors in estimating the angle foreach of the 100 grid points where the HRF was calculated. The box ineach column shows the 25th and 75th percentiles of the distribution.This experiment shows that modeling the HRF at each location of thelenticular array leads to nearly all angular measurements being within0.5 degrees of the true incident angle.

The errors in estimating angle from hue were evaluated to depend on theangle at which the calibration object was observed. FIGS. 14A and 14Bcomputed the distribution of errors across the entire array for eachimage angle. Again, the error is consistently small, even though thesestatistics are computed using anchor points that were substantiallyperturbed.

An error of 0.25° is near the limit of a simple geometric constraintbased on hue measured at one pixel. The lenticular array shows colorsacross the hue spectrum over a range of about 40°, so 0.25° is less than1% of the range of angles that are viewed. Reliably measuring the hue ofpixels in 8-bit RGB images to better than 1% precision was alsochallenging.

Example 4—Pose and Focal Length Estimation

In a laboratory setting, the performance of rotation was assessed alongwith translation and focal length estimation across differentviewpoints. On a motorized stage, the calibration object was rotated inincrements of 5 degrees from −25 to 25 degrees around the vertical axisand images were taken at each increment.

FIGS. 15A-15D show the rotation estimation performance per image in 15Aand 15C as well as in summary in 15B and 15D. The rotation error foreach local axis is shown as the angular difference of estimates to thetrue rotation. The estimates from the initialization algorithm are shownin 15B and show errors at the scale of a few degrees. FIG. 15D showsresults after minimizing the reprojection error, with rotation estimateswith a median error of 1 degree.

FIGS. 16A-16B quantified errors in the focal length estimation, andFIGS. 17A-17D quantified errors in the translation estimation. Both theinitialization and refinement results showed strong correlations betweenthe focal length error and the translation error. The refinement stepreduced the error of both to a median error of about 4%. The correlationin error between the focal length and the translation arises from theambiguity that an object can appear bigger either by moving closer tothe camera or by the camera changing its focal length.

FIGS. 18A-18H showed quantitative results for focal length estimationfrom single images of the calibration object taken at differentorientations and different focal lengths. For each image, the resultsshowed visualize rotation by rendering the local coordinate system ontop of the original image. The image title showed the ground truth focallength, the estimated focal length, and the percent error. Images fromcell phone cameras were included, as well as a DSLR camera. The firsttwo images are from an iPhone 5 and a Galaxy S6 with focal lengths of 5and 5.8 mm, respectively. The images following those are from a NikonD90 at focal lengths of 18, 49, 90, 115, and 185 mm.

Focal length estimates were relatively accurate for shorter focallengths. Very long focal lengths corresponded to imaging geometries witha smaller field of view. For small fields of view, small errors inestimating angular constraints may lead to larger errors in estimatingfocal length. To ground this, the impact of misestimating the field ofview by 0.25° was shown on the estimate of the focal length.

Example 5—Augmented Reality Application

In a desktop scene, video of the calibration object was recorded whilemoving the camera in a freehand trajectory. When the camera was movedfarther away from the scene and the calibration object, digital zoomingkept the calibration object as large as possible in the image. For eachframe, the camera focal length, rotation, and translation were estimatedusing the calibration object. In FIG. 20, the estimated focal length wascompared with the ground truth focal length per frame. It is seen thatthe focal length estimations follow the zooming trajectory well. It isemphasized that the algorithm did not have access to the digital zoominformation.

An AR algorithm was compared that doesn't have access to the digitalzoom and does not have the ability to estimate it from image data. Whensuch an algorithm used a pre-calibrated focal length, which became wrongin part of the video sequence, virtual objects were rendered withincorrect perspectives. FIGS. 21A-21I show 3 frames from the video ineach column. A virtual wireframe box was rendered to highlightperspective effects. FIGS. 21A-21C show the original images, FIGS.21D-21F show the box rendering given the estimates made with a dynamicfocal length, and the FIGS. 21G-21I show the box rendering given theestimates made with a static focal length. The digital box had a basethe size of the calibration object and was 45 mm deep.

Graph paper was aligned to show a Cartesian coordinate to help a viewerassess perspective effects. The wireframe box appeared aligned justshort (10 mm or 2 boxes) of the end of the paper grid. In comparing themethod of estimating a dynamic focal length against estimating a staticfocal length, it was seen that the rendered box looked unnaturally toolarge and with too much perspective in the case of a static focallength. This held true in general for all frames.

An additional AR video was also recorded. In this video, a 3-D model wasrendered into a video with a free-hand camera zoom.

Example 6—Anchor Point Permutations and Minimum Measurable Angle Change

FIG. 22 shows the anchor points used in calibrating the HRF as bluecrosses and the random perturbation as green crosses. The angularprecision resolvable of a lenticular array in an image is limited by thecolor sensitivity of the camera and color precision of color printing.This section analyzed the theoretical angular limits of a system imaginga lenticular array using an 8-bit RGB camera. The implemented lenticulararrays had their backplane textures printed at maximum saturation andvalue. The saturation/value of the lenticular array's appearance to thecamera, however, was determined by the amount of light in the scene andcamera properties such as exposure. The amount of light affecting theangular measurement precision was explored in this example.

The set of RGB values corresponding to the hue wheel for a givensaturation/value level lay on a 2-D manifold in RGB space. This manifoldrepresented the set of RGB measurements a camera would take of alenticular array in any orientation. In FIGS. 23A and 23B, two views ofthese manifolds are shown for various levels of saturation and value.The manifolds created a cycle along the sides of a cube aligned with RGBspace. Interestingly, as the amount of light (saturation and value) wentdown, the set of RGB values corresponding to the hue wheel got smaller.

In FIG. 24A, the number of unique 8-bit RGB triplets for each manifoldare shown, or each saturation/value level. The set of 1-to-1 view pointsof the lenticular array (76 degree range of incident angles) was mappedto the hue wheel and therefore to the RGB manifold. Thus, the angularprecision is 76/the number of unique rgb values. In FIG. 24B, theangular precision for each level of saturation/value was shown. In thebest possible case with maximum saturation and value, an 8-bit RGBcamera was able to resolve the angle of a lenticular array at aprecision of 0.05 degrees. However, at 0.3 saturation/value, theprecision dropped to 0.55 degrees. The hue measurements had a meansaturation and value of 0.7 and ranged from 0.5 to 0.9. Therefore, theangular error of 0.25 degrees induced by moving anchor points was notdue to the inherent precision limitations of the 8-bit RGB cameraimagining the lenticular arrays.

The angular precision achievable by a camera can be greatly improved bymoving to a larger color representation. In FIGS. 25A and 25B, the sameexperiment was run but for a 12-bit camera. With 16 times more possiblevalues for a single color channel versus an 8-bit camera, the number ofunique RGB values for the color wheel and the angular precision bothimproved by an order of magnitude.

Example 7—Augmented Reality Video

The approach was demonstrated with a second AR video where the camerawas static with a varying zoom, and the calibration object was beingrotated randomly. In this video, the zoom was achieved via a zoom lens.In one video, the wire-mesh of a box was overlayed to compare dynamicfocal length estimation versus static focal length estimation. In asecond video, instead of a box wiremesh, a 3-D model of a parrot wasoverlayed over the frames of the image. In FIGS. 26A-26F, frames of thisvideo are shown. Just like in the previous results, the dynamic focallength estimation ensured that the 3-D model was rendered with thecorrect perspective, no matter the zoom level.

What is claimed is:
 1. A method of calibrating a camera, the method comprising: creating a light field with a lenticular array, the lenticular array comprising at least one sheet, the at least one sheet comprising a plurality of colors; and, using a single picture of the light field to calibrate the camera, wherein the camera is calibrated without information from a correspondence between a point on an image and a point on a calibration object.
 2. The method of claim 1, wherein the plurality of colors are visible at a plurality of locations on the sheet.
 3. The method of claim 1, wherein the sheet comprises plastic.
 4. The method of claim 1, wherein the at least one sheet comprises a plurality of parallel cylindrical lenses.
 5. A method of calibrating a camera for an augmented reality application, the method comprising: using a calibration object to create a color-coded light field, wherein the calibration object is based on at least one lenticular array, each lenticular array comprising at least one sheet, the at least one sheet comprising a plurality of colors; estimating a focal length of the camera and a relative orientation and position of the object from a single image; and, calibrating the camera, wherein the camera is calibrated without information from a correspondence between a point on an image and a point on a calibration object.
 6. The method of claim 5, wherein the plurality of colors are visible at a plurality of locations on the sheet.
 7. The method of claim 5, wherein the sheet comprises plastic.
 8. The method of claim 5, wherein the at least one sheet comprises a plurality of parallel cylindrical lenses. 